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ABSTRACT 


Modern Test and Evaluation has long supported 
acquisition of warfighting systems in the United States 
Navy. As the complexity and long-term supportability of 
these systems has dramatically increased, the need to 
successfully, and incrementally test and evaluate families 
of systems, including their interfaces, has become even 
more critical. Long established techniques and 
methodologies for T&E may still apply, but new factors must 
be addressed. As the Navy continues to grapple with 
acquisition reform, and aims to transform itself in the 
future, the Warfighters' needs have essentially remained 
the same - delivery of the best, most effective weapons, as 
soon as possible, and made easy to operate and maintain. 
Without an equally effective developmental and operational 
test and evaluation process, the United States Navy cannot 
satisfy this need. 

This thesis examines T&E today and where it must go in 
the future. It provides recommendations for T&E 
enhancements, and explores several areas where the Navy, 
and Joint Services, is already looking towards future, 
integrated and collaborative test and evaluation. 
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EXECUTIVE SUMMARY 


Test and Evaluation (T&E) is required by law and 
contract for all major Department of Defense (DOD) 
acquisition programs. The science of T&E is currently 
taught to a portion of the Defense Acquisition Workforce, 
in the career field of T&E. The culture of T&E is embedded 
in the corporate history of all those who struggled to 
defend the need to both test and evaluate complex, critical 
weapons and combat systems being developed and fielded by 
the United States Department of Defense. As the DOD 
continues to transform itself, so must the T&E community 
keep up with the many challenges of this transformation, 
including the advent of evolutionary acquisition (spiral 
and incremental development) and development in an open 
architecture environment. 

This paper strives to provide a stamp in time of what 
the T&E community has been doing, what it is currently 
doing, and what can be done in the future to keep pace and 
to ensure that weapon systems acquired on behalf of every 
U.S. taxpayer are tested and evaluated in a manner that 
will deliver these weapons to our warfighters as quickly 
and efficiently as possible. It is also the responsibility 
of this same community to assure that the systems delivered 
are the best possible and will protect the lives of those 
on the front lines. And given rapid deployment of these 
weapon systems, we must ensure these systems perform as our 
soldiers, sailors and airmen need them to. 
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I. INTRODUCTION 


A. BACKGROUND 

Systems have been tested in the United States since 
the first weapons were developed for this country's use in 
defending itself, however modern testing could be 
associated with the advent of nuclear energy. The nuclear 
weapons age began on July 16, 1945 when the U.S. exploded 
the first nuclear bomb, codenamed 'Trinity' at Alamogordo, 
New Mexico. The "thermonuclear age" began on November 1, 
1952 when the U.S. exploded the first thermonuclear bomb at 
Eniwetok atoll in the Pacific. Codenamed 'Mike', this bomb 
was 500 times more powerful than the 'Trinity' test and had 
an estimated yield of 10.4 megatons. 



Figure 1. Nuclear bomb test Priscilla, June 

24, 1957 

According to the environmental lobby Greenpeace, the 
U.S. has carried out 1,030 nuclear weapons tests (the last 
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and final test on 23 September 1993); the equivalent of one 
nuclear weapons test every 17 days since its first test 
(Campaign History, 2002.) These test were planned to be 
successful. Each test was a step in an overall master test 
plan that would guarantee success of the program while 
maintaining a broad enough region of uncertainty to 
compensate for the unexpected. These were extremely 
regimented programs. While certainly a formidable 

challenge to appropriately test nuclear weapons, this paper 
will focus on conventional (non-nuclear) Department of Navy 
(DON) weapon systems under DOD development. 

For the purpose of this research, and as defined in 
the original version (dated December 1996) of SECNAVINST 
5000.2B: 

A "weapon system" is an overarching term 
that applies to a host platform (e.g., ship, 
aircraft, missile, weapon, combat system 
subsystem(s), component(s), equipment(s), 

hardware, firmware, software, or item(s) that may 
collectively or individually be a weapon system 
acquisition program (i.e., all programs other 
than information technology programs). 

B. PURPOSE 

The purpose of this research is to provide a 
historical account of what has been done in the past, what 
is currently being accomplished, and what could be done in 
the future to ensure that every weapon system acquired on 
behalf of U.S. taxpayers is tested and evaluated in a 
manner that will deliver these weapons to our warfighters 
as quickly and efficiently as possible. And given rapid 
deployment of these weapon systems, DOD must ensure they 
work as our soldiers, sailors, and airmen need them to. 
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This research will examine several factors that should 
prompt an evolution in how modern T&E must be conducted. 
T&E must continue to support the many DOD weapon systems 
under acquisition at present, and within the coming decade, 
but it must be agile enough to accommodate future, open 
weapon systems, which will have potentially different sets 
of requirements and risks to be weighed only through 
conscientious and appropriate testing and evaluation. 

C. RESEARCH QUESTIONS 

The intent of this research study is to focus on a 
variety of questions, some in depth, and others less so, to 
build a case that test and evaluation (T&E) as it is 
conducted today, must evolve to keep pace with the DOD as 
it undergoes reform, transformation, perpetually shifting 
requirements, budget fluxuations, and an emerging and 
dangerous new set of enemies and unforeseen threats. This 
set of questions can be grouped into four themes, including 
history and the present, guidance and leadership, open 
systems, and T&E in the future. 

History and the Present : 

• How are Navy weapon systems acquired today? 

• How are US Navy surface combatant weapon systems 
evolved today? 

• What is Test and Evaluation, and how is it conducted 
in today's Navy? 

Guidance and Leadership: 

• What is acquisition reform, and how does it apply to 
T&E? 

• What does Transformation mean with respect to T&E? 

• What do current Navy Leaders think about T&E today? 
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Open Systems : 

• What are "open systems"? 

• What is "open architecture" and what is the Navy's 
commitment to OA? 

• How will OA improve weapon systems in the future? 

• What are recent improvements to OA? 

• What are the advantages and disadvantages of OA? 

• What are examples of existing OA systems? 

T&E in the Future : 

• What is required to properly test and evaluate future 
Navy systems? 

• What are the recent changes in the methodology of 
weapon systems computer program development? 

• How are Joint systems tested and evaluated? 

• What is evolutionary acquisition, and how does it 
apply to T&E of those future systems? 

• How should T&E be taught to ensure future T&E 
professionals would be prepared for future challenges? 

• How must T&E evolve in the future? 

It should be noted that the AEGIS program 
(specifically the AEGIS platform, the AEGIS Weapon Systems, 
and the AEGIS Weapon System Computer Program) will be used 
extensively as a case study when exploring many of the 
questions stated above. In addition, some attention will 
be focused towards the Missile Defense Agency, however 
mainly as it relates to the AEGIS Ballistic Missile Defense 
(ABMD) development effort. 
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D. POTENTIAL BENEFITS FROM THIS STUDY 

As a member of the T&E acquisition workforce, and a 
T&E practitioner for approximately the last 15 years, the 
author's sincere hope is that there will be several 
benefits from this research study. This research shall 
provide recommendations and assessments to both DON and the 
T&E professional acquisition workforce on what can be done 
to prepare for testing of future, open systems. 

In addition, this research is hoped to have actual 
benefit to the Warfighters of the future, who will depend 
on timely and appropriate testing and evaluation, leading 
to weapons on target, and the ability to fight and win, 
unhampered by systems which offer technology, but are not 
suitably tested and ready to go into harms way. 

E. SCOPE AND METHODOLOGY 

1. Scope 

The scope of this research study is divided into five 
parts. The first part is the introduction, and includes a 
brief discussion on DOD acquisition, acquisition reform, 
T&E, and what open systems means to DOD and how the rush to 
get state-of-the-art, open systems to the Warfighter, 
presents a unique set of challenges to both testers and 
evaluators. 

The second part involves a historical review of test 
and evaluation, touching on the acquisition reform 
discussion from the background section, but going into more 
detail. This section will expand on the history of T&E, 
T&E guidance, and T&E in practice today. This section will 
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also include a short discussion about where T&E needs to go 
in the future, which is expanded in much greater detail in 
section four. 

The third part will focus on open systems and open 
architecture, including current guidance as related to DOD 
systems under acquisition today and standards for open 
architecture, applicable to weapon systems to be acquired 
in the future. This part will also discuss a few ongoing 
examples of system under current development, including the 
advantages, as well as challenges, of working with open 
systems. 

The fourth part will use the findings from section 
three, to build a case for T&E in the future. 

Finally, the fifth section will present conclusions 
and recommendations for further study. 

The end result from this research is to contrast where 
modern T&E appears to be headed in the future and where it 
needs to go based on the latest published acquisition 
reform guidance, and based on where open systems 
development will effect future DOD development and future 
weapon systems undergoing T&E. 

2. Methodology 

The methodology used in this research consists of the 
following: 

• Conduct a literature review of DOD and DON related 

guidance and reports on T&E and acquisition reform. 

• Conduct an in-depth review of available Program 

Executive Office (PEO) level briefings and white 

papers covering acquisition reform, transformation. 
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and steps to address legacy systems, either in-service 
or currently under development and acquisition today. 

• Interview members of the Acquisition Workforce, 
specifically. Test and Evaluation Professionals to 
assess their efforts to prepare for emerging, open 
systems to be developed. 

• Interview members of various Program Executive 
Offices, who are presently involved in the acquisition 
of systems which will be "open" from the inception to 
assess their opinions on how prepared we will be to 
test and evaluate their systems in the future. 

• Participate in T&E communities of practice, including 
the International Test and Evaluation Association 
(ITEA), and the Defense Test & Evaluation Professional 
Institute (DTEPI). 

• Conduct in-depth Internet research on all topics to 
determine what information is in the public domain and 
to determine how commercially produced, open systems 
are tested and evaluated today. 

F. ACQUISITION TODAY 

Defense acquisition's primary objective is to obtain 
cost-effective, quality weapon systems, in a timely manner, 
while meeting an operational need. Today's modern 

warfighting systems are acquired under a series of DOD 
instructions, directives and regulations. The Secretary of 
the Navy implemented SECNAVINST 5000.2B in December of 1996 
to provide a framework for mandatory procedures applicable 
to all major and minor DON acquisition programs. 
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Acquisition policy continues to evolve under what has 
commonly been referred to as acquisition reform. Even when 
SECNAVINST 5000.2B was written over seven years aqo, its 
authors understood that acquisition would need to evolve 
further. In fact, the instruction referenced a term that 
would become a catch phrase for modern acquisition 
"Evolutionary Acquisition." 

As stated in SECNAVINST 5000.2B, "When an evolutionary 
acquisition (EA) strategy is used to field a core 
capability and there are subsequent modifications to the 
initial fielded core capability, such modifications shall 
satisfy a validated requirement and be supportable in the 
operational environment. EA modifications to the core 
capability shall be funded, developed, and tested in 

manageable increments. Each increment shall be managed as a 
modification ." 

Recently, the Secretary of Defense, Donald Rumsfeld 
called for a new Department of Defense acquisition system. 
In January of 2001 during a nomination hearing before the 
Senate Armed Services Committee, Secretary Rumsfeld 
(Canahuate, 2001, 12 ) said, "The present weapons systems 

acquisition process is ill-suited to meet the demands posed 
by an expansion of unconventional and asymmetrical threats 
in an era of rapid technological advances and pervasive 
proliferation." Later that same year, in October of 2002, 
Deputy Defense Secretary Paul Wolfowitz cancelled all 
existing acquisition rules, and stated that new ones should 
be prepared. At the time, he provided interim guidance 

pointing to a simpler system to "rapidly deliver 

affordable, sustainable capability to the warfighter that 
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meets the warfighter's needs. (Caterinicchia, 2003) 
Further, Secretary Wolfowitz' memo, spoke of "transforming" 
the military, and Defense Secretary Rumsfeld has urged 
civilian and military leaders to acquire new, high-tech 
systems. And once acquired, these systems must be rapidly 
delivered onto the battlefield. 

DOD acquisition still faces challenges. In his March 
of 2003 resignation letter to President Bush, 
Undersecretary of Defense for Acquisition, Technology and 
Logistics, Edward "Pete" Aldridge Jr. summarized his top 
five goals for achieving "acquisition excellence" within 
DOD: 

1) Improve the credibility and effectiveness of the 
acquisition and logistics support process. 

2) Improve the morale and quality of the acquisition 
workforce. 

3) Improve the health of the defense industrial 
base. 

4) Support the decision process by rationalizing 
weapon systems and defense infrastructure with 
the new defense strategy. 

5) Initiate high-leverage technologies that would 
provide the war-winning capabilities of the 
future. 

"All in all I think we have made significant progress 
on accomplishing these five goals and setting in place the 
acquisition, technology and logistics support activities 
that you and Secretary [Donald] Rumsfeld want to have for 
DOD," his letter said (Caterinicchia, 2003, 54.) 


9 



G. 


EVOLUTIONARY ACQUISITION - EXAMPLE: MDA 


The Missile Defense Agency (MDA), 
released the following information 
acguisition strategy (BMD Fiscal Year 2004 


in July 2003, 
regarding its 
Budget, 2003, p. 


2 .) 


MDA is following an evolutionary acquisition 
strategy for the BMD System that effectively 
manages changes in the threat, changes in BMD 
System technologies, and progress in development 
and testing. Using Research, Development, Test 
and Evaluation (RDT&E) resources almost 
exclusively and in conjunction with an 
evolutionary approach, the strategy capitalizes 
on technological progression and provides for 
development, limited production, and deployment 
of initial BMD capabilities incrementally as soon 
as they are ready. Adopting an evolutionary 
acquisition model, the BMD System is constructed 
around a "Capability-based Block" approach. Each 
BMDS Block spans a two-year timeframe and 
continuously builds capability into the BMD 
System by introducing new sensor and weapon 
projects, and/or by augmenting and enhancing 
existing capabilities. As the new projects mature 
they will be integrated into the BMD System to 
increase the capability to respond to the 
evolving threat. BMDS Block management includes 
decision points at which activities will be 
evaluated on the basis of effectiveness within 
the overall system, technical risk, deployment 
schedule, and cost. From these decision points, 
developmental activities will be accelerated, 
modified, or terminated depending on progress and 
promise. 


H. TEST AND EVALUTATION 


Two broad types of testing, which will be discussed in 
this document, are used to assist DOD in meeting the goal 
of defense acquisition, as laid out in section I.F. 
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Developmental testing covers a wide range that includes 
component and systems engineering testing, as well as 
modeling and simulation. Developmental testing affords the 
first chance to assess performance and effectiveness of a 
weapon system against tolerances laid out in the analysis 
of alternatives. Operational testing will focus on 

performance of a fully integrated set of systems, ideally 
within a realistic operating environment. Testing at the 
operational level is the process by which DOD assesses 
whether a weapon system can satisfy planned capability 
before deciding to begin full-rate production. In 

addition, operational T&E uses independent assessment to 
determine if a system is effective and suitable for its 
particular application. 

DT&E is required for all developmental acquisition 
programs. For DON programs, the Design Agent (DA) through 
contractor testing or government test and engineering 
activities shall conduct DT&E. Combined developmental 
testing/operational testing (DT/OT) shall be pursued 
whenever possible to reduce program costs, improve program 
schedule, and provide early visibility of performance 
issues. 

The DOD Under Secretary for Defense Acquisition, 
Technology and Logistics (USD (AT&L)) / Defense Systems 

maintains a staff element responsible for assuring that 
DT&E programs are sound, well-executed and sufficiently 
address the modern warfighters' needs. USD (AT&L) refers 
to this group as Developmental Test & Evaluation. Their 
mission (DOT&E Mission Statement, 2003, p.l) is to ensure 
development of sound and well-executed test strategies 
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within DT&E programs. 

and 

to ensure 

that DT&E matures a 

program; 

allowing it 

a 

good chance 

of achieving 

it's 

critical 

operational 

design goals. 

This group 

also 

provides 

the focal point 

for DT&E 

policy under 

United 


States Code Title 10, Section 133. 


As 

part 

of 

the 

T&E Best Practices 

Conference, 

sponsored 

by 

DT&E 

USD 

(AT&L), 

"T&E ensures 

our weapon 

systems 

perform 

as 

desired 

and meet 

warfighters' 


requirements. (Weapon systems must) work when and how they 
are supposed to."(Lockhart, Richard, Integrating Test and 
Evaluation, 2002) The role of T&E in the acquisition 
process is: 

• Provide essential information on which to base 
acquisition decisions. 

• Assess technical performance and system maturity. 

• Provide indication of program's development 

progress. 

• Provide information about risk and risk 

mitigation. 

• Identify problems so they can be resolved early. 

• Confirm weapon system's readiness to enter IOT&E. 

• Advise on how best to use the system. 

• Confirm weapon system meets user requirements. 

I. CHALLENGES TO THE DOD 

The DOD will always face major management challenges 
and program risks as it seeks to support and defend the 
Constitution of the United States; provide for the common 
defense of the nation, its citizens, and its allies; and 
protect and advance U.S. interests around the world. 
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In the latest report to Congress ("Major Management 
Challenges and Program Risks," 2003, p. 5) the GAO 
summarized these challenges into eight areas. Nearly all 
of the major challenges to DOD apply directly to Test and 
Evaluation. From the need to hire, train, sustain and 
maintain a T&E workforce, to having the necessary 
infrastructure (ranges, targets, services, etc.) in place 

to engage in meaningful T&E, to having proper budgets, and 
using technology, to keeping a mindful eye towards costs 
effectiveness and timeliness, and monitoring and reducing 
risk, it seems T&E's major challenges are simply a subset 
of what DOD needs to do to improve and complete its 
fundamental mission. The goal for the T&E community should 
be to help DOD along this continuous path of improvement. 
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Performance and 
Accountability Challenges 



• Strengthen strategic planning and budgeting to achieve desired mission 
outcomes 

• Hire, support, and retain military and civilian personnel with the skills to meet 
mission needs 

• Overcome support infrastructure inefficiencies to reduce costs and improve 
operations 

• Confront and transform pervasive, decades-old financial management 
problems to improve financial accountability 

• Effectively manage infoimation technology investments to transform business 
functions 

• Improve DOD's ability to acquire weapon systems in a cost-effective and 
timely way 

• Improve processes and controls to reduce contract risk 

• Provide logistics support that responds to the needs of the warfighter at an 
affordable cost 


Figure 2. 


DOD Major Challenges 


(2003) 
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II. T&E: A HISTORICAL VIEWPOINT 


A. LEADERSHIP PERSPECTIVE 

Steadfast leadership by program development managers 
has brought us to where we are today. During the early 
days of nuclear power, without the strong and unfailing 
conviction of Admiral Hyman Rickover, who once said, "Good 
ideas are not adopted automatically. They must be driven 
into practice with courageous patience," the nuclear Navy 
might never have emerged into the uncontested force it 
remains today. 

During the early days of Surface Missile Ship 
development, when existing weapon system were thought to be 
sufficient to meet the threat, RADM Wayne E. Meyer 
aggressively pushed for a fully-integrated weapon system. 
His "build a little, test a little, learn a lot," approach 
allowed the DON to field the most capable surface combatant 
ever. 


AEGIS PHILOSOPHY 


BUILD-A-LITTLE 


TEST-A-LITTLE 


LEARN-A-LOT 


Figure 3. AEGIS System Engineering 
Philosophy (1977) 
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The Honorable William J. Perry, Secretary of Defense 
from 1994-1997 stated, "testing is the conscience of 
acquisition." 

B. TEST AND EVALUTATION - IN THE NAVY 


POD T&E Organization 



Sec Army 
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DUSA(OR) 
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Commandant, 
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Naval Ops 

i) 


Army 
Test and 
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Command 
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Marine Corps 


Marine 

Systems 


Corps 

Command 


Operational 

(MARCORSYSCOM) 


Test and 


Evaluation 


Activity 


(MCOTEA) 


Navy 


Operational 

Systems 


Test and 

Commands 


Evaluation 

(SPAWAR) 


Force 

(NAVAIR) 


(OPTEVFOR) 

(NAVSEA) 



Air Force 


Air Force 

Material 


Operational 

Command 


Test and 

(AFMC) 


Evaluation 


Center 


(AFOTEC) 


NOTE: Other Defense Components (e.g. DFAS, DLA, DISA, etc..) are also subject to rules and regulations governing Test & Evaluation 


Figure 4. DOD T&E Organization (2003) 

As defined in SECNAVINST 5000.2B, the following 
guidance is provided with respect to T&E: "Early 

involvement between the developing activity (DA) and the 
operational test agency (OTA) Operational Test and 
Evaluation Force (OPTEVFOR)/Marine Corps Operational Test 
and Evaluation Activity (MCOTEA) is required to ensure that 
both have a common understanding of the proposed system 
requirements and that developmental and operational testing 
is tailored to optimize cost, schedule, while verifying 
performance. The Commander, Marine Corps Systems Command 
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(COMMARCORSYSCOM) and Director, MCOTEA are the principals 
responsible for developmental test and evaluation (DT&E) 
and operational test and evaluation (OT&E), respectively, 
within the Marine Corps. MCOTEA is designated as the Marine 
Corps independent operational T&E activity responsible for 
adequate testing, objective evaluation, and independent 
reporting in support of the Marine Corps Acquisition 
Process. 





OPERATIONAL TEST 
DIRECTOR’S GUIDE 

WuMANDER. OPERATIONAL TENT AND 
H EVALUATION FORCE 

NORFOLK VIRGINIA 


Figure 5. COTF OTD's Guide (2001) 

The Operational Test Director's Guide is an 
instruction published and maintained by COMOPTEVFOR for the 
benefit of OTD's, OTC's and is a valuable reference for the 
entire DON T&E community. The OTDG is designed to provide 
the operational test director with guidance on the various 
aspects of operational test and evaluation. 

C. TEST AND EVALUATION - AEGIS 

1. Leading Up to AEGIS 


During the World War II era, aircraft attacks against 
naval ships became a common threat. It became evident 
during this time that using small (20mm and 40mm) and 
medium (3" and 5") caliber man crew-served weapons against 
enemy air threats was not adequate to defend against the 
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threat. In addition, technological improvements in 
aircraft design overwhelmed current-day Anti-Aircraft 
capabilities of US naval defense systems. Foreign nations 
were creating faster, more maneuverable aircraft, more 
difficult to counter, and requiring greater manpower. 
Towards the end of WWII, a new threat was introduced when 
the first successful air-to-surface missiles became 
available. These computer-controlled missiles demonstrated 
vulnerability of the surface ships due to precision 
attacks. As a controlled experiment the Johns Hopkins 
University Applied Physics Laboratory (JHU/APL) started 
developing surface-to-air guided missile capabilities on 
TERRIER, TARTAR, and TALOS (AKA "3T") systems (Lundquist, 
2002, St 33.) The TALOS Fire Control System was a long 
range Beam-Rider system deployed on larger vessels 
including refurbished WWII vintage Cruisers. Second was 
the TERRIER system; a medium range fire control system 
deployed on smaller DLG's, (Large Destroyer/Light Cruiser). 
Finally was the TARTAR fire control system, a short-range 
missile system deployed on USS Adams DDG-2 class 
destroyers. These were truly the first ships specifically 
designed to handle a missile fire control system. When 
developed, the 3T's were intended as direct replacement for 
existing anti-air gun system. The radar system on these 
ships reported range, bearing, and elevation data, which 
was input to an analog computer that determined the range 
of open fire and generated the necessary orders for 
launching a missile into the radar guidance beam. The 
radar guidance beam guided the missile and developed the 
necessary steering instructions to intercept its target. 
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This phenomenal technology later became the basis for the 
state-of-the-art SPY radar system on current AEGIS Cruisers 
and Destroyers. 

The standing Chief of Naval Operations, ADM Arleigh 
Burke, had recognized the need for surface-to-air missile 
systems with performance capabilities greater than the 
inherent capability of the 3T. Projected advances in 
threat speeds and the ability to be subject to coordinated 
mass attacks would still require an even faster reaction 
time and greater firepower resulting in a technologically 
advanced system to be called, "Typhon". Human reaction 
time was no longer sufficient to defend against attacks of 
larger magnitude and speed, and a computer-driven system 
was the answer for faster reaction time. The Bureau of 
Naval Weapons initiated Typhon in 1960. Most of the 
efforts during the development and testing of the Typhon 
system was to revolutionize the new radar system that was 
developed by the JHU/APL. The Typhon program developed 
principles needed to effectively build a more advanced 
weapons system. However, insufficient attention and 

emphasis were afforded to operational suitability and 
support requirements. The state-of-the-art technology 
available at the time was still too primitive to achieve 
the performance goals sought within appropriate size and 
weight requirements. Lessons learned from the Typhon 
project led to planning inception of the AEGIS program in 
1963 (Madsen, 1986.) 

By the late 1960's, the United States Navy realized 
that reaction time, firepower, and operational availability 
in various environments were not sufficient to counter 
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increasingly sophisticated enemy threats. The U.S. Navy's 
inability to adequately defend itself called for the 
proposal of a more advanced defense system. The Advanced 
Surface Missile System (ASMS) program began in 1965. 
Secretary of the Navy led a comprehensive engineering 
development program for ASMS. Following the cancellation 
of the Typhon program, the ASMS project was launched, later 
renamed AEGIS in 1969. AEGIS, which is a term used for the 
armor of Zeus (hence the phrase "under the aegis of" or 
"under the protection of"). Integrating still-evolving 
state-of-the-art radar and missile systems, AEGIS is a 
complete system designed to handle tactical engagements 
from detection through kill assessment. Designed as a 
fully integrated weapon system, it was built to defend 
against advanced air, surface, and subsurface threats. The 
AEGIS computer program is a set of operations controlling 
and operating the entire combat system. The AEGIS Weapon 
System computer program provides a fully automated response 
to threats (via selection and application of doctrine 
parameters), normal operation of Command and Decision (CND) 
process, and on-line system performance assessments. There 
are about one million words (using CMS-2Y language) in the 
computer programs for the AEGIS computer system using the 
UYK (General utilities Data processing Computing) 
processing unit. 

2. Test Methodology 

Prior to the introduction of the first AEGIS Cruiser 
the United States Navy had already developed a methodology 
for test and evaluation of Missile Fire Control Combat 
Systems. There were three distinct Missile Fire Control 
Systems deployed with fundamental differences between each. 
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First was the TALOS Fire Control System; the long-range 
beam-riding system deployed on larger vessels including 
refurbished WWII vintage cruisers. Second was the TERRIER 
system; a medium-range fire control system deployed on 
smaller DLG's, (Large Destroyer/Light Cruiser). The third 
was the TARTAR fire control system; a short-range missile 
system deployed on USS ADAMS (DDG-2) class destroyers. 
These were the first ships specifically designed to handle 
a missile fire control system (Lundquist, 2002, St 33-35.) 



From the 1960's through the early 1980's, these ships 
were the front line sea defense for the Navy's carrier 
battle groups. Most of these systems directly supported 
operations during the Vietnam Conflict and TERRIER and 
TALOS systems were credited with kills of hostile aircraft 
during that timeframe. 
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However these systems were designed with out-dated 
technology, requiring an extensive amount of maintenance to 
remain operational. Included in this maintenance was the 
assembly of missile parts prior to positioning on the 
launcher rail for firing. Computers were analog, (syncros, 
servos, gears and other discrete components) and required 
constant adjustment and alignment to maintain material 
readiness. It was not until the late 1970's, when the 
Navy's MK-1219 digital computer, (the first digital missile 
fire control computer) was retrofitted into existing 
systems. Insertion of technology was done similarly for 
years to come. 

Testing directly on the platform, during major 
upgrades and revisions became the foundation of the test 
and evaluation process for many years to come. This 
philosophy was based on emerging technology and an ever- 
evolving threat. 

Weapon systems, new and old, had to be maintained and 
tested. Each ship was evaluated against minimum standards 
to determine battle readiness. These "first generation" 
missile-guidance ships were evaluated very much like 
earlier ships, except that special tests were included to 
verify performance of the fire control system. 

Each ship was evaluated first on maintenance. These 
early missile systems required a significant level of 
maintenance, which kept the crew very busy. During 
evaluation periods, maintenance actions were randomly 
inspected along with the maintenance paper work and 
documentation as well as crew training to perform the task. 
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For ships undergoing new construction, or coming 
through a major overhaul period, (every three years at that 
time), a Refresher Training, (REFTRA), and Combat Systems 
Ships Qualification Test, (CSSQT) were added to the ships 
schedule. CSSQT could be described as the first and only 
end-to-end combat system training. These training and test 
evolution periods had to be completed before deployment, 
and failure due to lack of training or system deficiency 
was a serious matter. Standards of evaluation were very 
high for these Naval Combatants. These ships were 

independently tested and integrated units that would later 
be required to operate in a battle group. 

3. At-Sea Testing 

During new construction or ship refurbishment, 
components of the new weapon system are assembled and 
integrated for the first time when they are installed on 
the ship. Previously, each element had been individually 
tested in the factory, where each piece of equipment met 
standards of construction and performance. This was the 
only insurance that these components would integrate 
properly within a functional weapon system. 

Integration was carried out on the waterfront; a 
process where all the weapon system elements came together 
and were tested as a system for the first time. The 

measurement of this integration was conducted during a 
daily maintenance action called a DSOT, or Daily System 
Operability Test. This test included the generation of a 
simulated target, the assignment of radar, the training out 
of the launcher, and the loading of a test missile where 
firing voltage was applied and a tail cone light 
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illuminated if the circuit was complete. This test 
verified system performance, each day. 

FY 75 CONGRESSIONAL 
CRITERIA FOR AEGIS 

"The conferees concur in the fact that subsequent 

authorization requests for Aegis will be predicated upon: 

• Successful at-sea testing that demonstrates the ability 
of Aegis to meet its prescribed performance objectives... 

• At-sea operation and maintenance of the Aegis system 
by shipboard personnel... 

• A cohesive integration plan specifying the interface 
of Aegis with the platform(s) and other weapon and 
command/control systems... 

• Definition and approval by both the Navy and the 
Department of Defense of the platform(s) for Aegis...” 

CONGRESSIONAL RECORD - HOUSE 
JULY 24. 1974 

Figure 7. Congressional Criteria for AEGIS 

(FY7 5) 

Integration on the waterfront was costly, requiring the 
numerous support engineers and shipyard workers to make it 
all work. The effort also took time, but proved to be the 
most effective way to integrate weapon systems in those 
days. Due to the many unique features of each respective 
ship, waterfront integration became a "living process." 
Within this process was conceived the notion that 
performance of a "Class Of Ships" could only be realized 
based on "Individual Hull" trend performance. This meant 
that class requirements could be applied, as milestones, 
for each individual ship to satisfy. However, compliance 
to these standards differed vastly from ship to ship. 
While some ships demonstrated superior tracking radar 


performance, others enjoyed stable launching systems. Mean 
Time Between Failure (MTBF) was dependent on crew training 
and aggressive diligence to maintenance procedures. 

Across all phases of integration and test, at-sea 
testing of ships remained the best measure of performance. 
Ships were not certified for deployment unless successfully 
completing REFTRA and CSSQT. CSSQT required live firing at 
a target. CSSQT, although centered on a single ship, 
involved the entire battle group during missile firing 
events. CSSQT could therefore set the stage for 
deployment, and provided insight into how the various ships 
might interoperate during tactical operations. At this 
stage, the Navy would use land-based testing to decide 
whether to purchase components for these new missile weapon 
systems, but for end-to-end system certification, at-sea 
tests was required. 

4. AEGIS Weapon System Development 

A principle design goal of the AEGIS Weapon System was 
to apply technology in such a way to build a new system far 
superior to that of currently fielded missile fire control 
systems. Weapon system complexity was a main challenge. 
In the earlier systems, computers had transitioned from 
analog to digital. The 1219 computer was limited to 64K 
Bytes. Computer programs for AEGIS were envisioned at 
millions of Bytes and the computer was completely 
different. The first AEGIS computer used was the AN/UYK-7 
in a 4-bay configuration, bringing computing power never 
before realized to the missile fire control system. To 
properly test this new AEGIS system, new methods of ET&E, 
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from the element level, to the system level, would have to 
be pioneered - a lofty challenge that still, to this day, 
is evolving. 

AEGIS integration and test was carried out at a number 
levels, closely monitored by the Navy's technical 
factory/manufacturer representative (TECHREP) at the 
various manufacturers that furnished AEGIS equipment 
components or assembled and tested AEGIS components. 


AEGIS WEAPON SYSTEM PRODUCTION 



Figure 8. AEGIS Development & Testing (1983) 


a) COMPONENTS - As specific pieces of the weapon 
system are being built, parts are constructed 
to DOD standards for manufacturing and 
reliability. Each component is tested once 
installed in the element piece. 

b) ELEMENT PIECES - Each circuit card was tested 
before installation into its respective 
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cabinet. Cabinets, when assembled, were tested 
individually to weapon system specifications. 

c) WEAPON SYSTEM COMPONENTS - Each weapon system 
component was assembled in a production test 
center (PTC) where integration and testing 
would be conducted at a "System Level" 
configuration. This testing was the basis for a 
level of performance that had to be duplicated 
at the shipyard for waterfront integration. 

d) COMPUTER PROGRAM - Unlike previous missile fire 
control systems, computer program testing for 
AEGIS had to be started in a land-based 
environment, with interfaces being simulated. 

After initial land-based testing was completed, 
where the program was checked out in tactical 
hardware resident at the land-based test site, 
the program was then delivered to the PTC, 
where it was installed into the actual tactical 
equipment that would be delivered to a specific 
ship. 

5. Land-Based Testing 

Computer program integration has evolved with the 
technology. The difficulty in performance verification is 
compounded when these dramatically more complex systems 
(that these programs are designed to control have) vary in 
configuration from ship to ship. Upgraded components and 
configuration corrections in support of an ever-evolving 
system, although not by design, ensured that each ship 
would be unique. Even though the "ship class" held to 
standards of performance, each ship would find subtle, and 
sometimes not so subtle, variations that would need to be 
addressed through crew training and proficiency. 

Land-based testing of the AEGIS computer program is 
currently conducted at several locations. The primary site 
for this testing was, and remains, the Combat System 
Engineering Development Site, (CSEDS), in Moorestown New 
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Jersey. This site was designed to house sufficient "end 
item" weapon system equipment to provide both system test 
and operator/crew training. 



Figure 9. AEGIS Combat System LBTS 
Development (1977) 

Another facility is the Production Test Center, 
located in Moorestown, New Jersey, where all of the 
individual components of the ships weapon system are 
assembled and tested, and computer program development and 
testing is executed. In the early stages of construction 
and "sell-off", confidence in the capabilities of the 
system is traditionally high. The methodology for 
development and deployment of the first AEGIS system was to 
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"build a little, test a little," a paradigm that has 
remained consistent into current-day testing. 

6. AEGIS At-Sea Testing 

Proven through history, land-based testing is not, by 
itself, sufficient to certify performance. At-sea live 
fire testing is required. During development of the AEGIS 
Weapon System, at-sea testing was required before the 
system was released for production. After extensive Land- 
Based testing at CSEDS, a pre-production system was sent to 
the USS NORTON SOUND, a converted WWII seaplane tender, and 
became the home of the first AEGIS Weapon System. The 
system was installed and a massive test program was 
initiated. At-sea operations were conducted in stand-alone 
modes and also with other naval units when opportunities 
allowed. Multiple live fire events were conducted and 
AEGIS eventually proved to be a capable and flexible 
replacement for the already aged TARTAR, TERRIER, and TALOS 
systems. 

After the release for production was given, the next 
phases of At-sea testing were completed during the new 
construction period at the shipyard. Prior to AEGIS, 
shipyard integration did not include a live missile-firing 
event. To this day, each AEGIS combatant is required to 
fire at least one missile during shipbuilder's trials prior 
to custody transfer of the ship to the Navy. Even in 
today's budget-conscious environment, builder's trials 
missile firings are mandatory for compliance to integration 
requirements. 

The T&E community is continually challenged to 
demonstrate regression performance has been assured from 
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ship to ship, and baseline to baseline. Each ship, once 
constructed, must pass the same CSSQT requirements that 
previous systems have been required to satisfy. As the 
complexity of each follow-on AEGIS system has grown, so 
must the level of testing that is completed during each 
CSSQT. 

AEGIS CSSQTs and live firing events have specific test 


objectives, with 

respective measures of 

performance 

that 

simply cannot 

be 

tested 

in a land-based 

environment 

and 

which often 

requires 

live ordnance 

to satisfy 

the 

obj ective. 

To 

ensure 

a high standard 

of testing 

is 


maintained, AEGIS test objectives are developed, approved, 
certified, and evaluated by the entire AEGIS technical 
community. 

Over the last 23 years, thousands of AEGIS live fire 
test events have been completed at sea. The AEGIS 
community maintains a controlled closed loop engineering 
process that monitors system improvements and makes sure 
that ET&E events are at a level sufficient to adequately 
test the performance of the system. Whereas every AEGIS 
ship commissioned by the Navy is quantifiably unique, 
testing of specific measures of performance is required on 
each and every ship of the class. 

7. Aegis in the Future 

As the AEGIS Weapon System evolved, the ship classes, 
which carried the TARTAR, TERRIER, and TALOS systems, were 
decommissioned. These early missile systems led to the 
development of newer systems, including AEGIS, and pushed 
the limits of what T&E could provide. The latest versions 
of the AEGIS Weapon System continue to evolve, and so must 
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the state of testing and evaluation. As today's threat 
evolves, so must future weapon systems. Testing, whether 
land-based, at-sea or via modeling and simulation, must 
continue until the last ship slips away and a newer ship 
takes on the roll of defender of the fleet. 

D. TEST AND EVALUTATION - LOOKING FORWARD 

The future of T&E should trace to the requirements and 
features of current and evolving threats, and in the 
designs and advanced concepts for future weapons and combat 
systems. For T&E to continue to provide the confidence and 
assurance in feasible, effective and suitable future 
systems, it must become more agile and more embedded in the 
process of acquisition. T&E consists of major and minor 
milestones across the acquisition timeline, which take time 
out, or away, from the program development. It is at these 
times that, ideally, the design must freeze, and in 
essence, a snapshot in time is of 80 taken in terms of 
performance and adequacy of design. Did we meet our 
specifications? Did we achieve expected tolerances? Did 
we get it right, in terms of where we are at along the 
development cycle? But in the future state percent 
solutions, and considering dramatically shrinking 
timelines, future T&E must be ingrained into the fabric of 
design and development. 

Further, the premise of operational testing, including 
evolving operator needs, must be considered. The impact 
from the realization that suitability and effectiveness of 
design has not been met is lost if the system has already 
been delivered to the warfighter. The warfighter is 
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resolved, in fact trained, to make these systems work. 
Testing early, and rigorously under precise and controlled 
conditions is often a given. However also factoring in 
operational conditions is key towards ensuring the system 
under development is right for the mission. 




Figure 10. Joint Vision 2020 

The Joint Chiefs of Staff recently built off the foundation 
established in Joint Vision 2010 and stated that Joint 
Vision 2020 should consist of dedicated individuals and 
innovative organizations transforming the joint force for 
the 21 st Century to achieve full spectrum dominance to be 
persuasive in peace, decisive in war, and preeminent in any 
form of conflict. Several new areas were highlighted in JV 
2020, including Joint Command and Control, 
interoperability, and Information Operations. 
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Table 1. T&E Career Field Developmental Guide 

(1999) 

determine whether systems are operationally effective, 
suitable and survivable for intended use. Further, the 

definition goes on to describe two types of T&E 

Developmental (DT&E) and Operational (OT&E). The latest 

release of the Program Managers Tool Kit has a helpful 
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diagram related to Test and Evaluation. This diagram 
compares and contrasts DT&E and OT&E, and provides a 
summary of production qualification T&E (sometimes referred 
to as PT&E) , live fire T&E (LFT&E) and initial operational 
T&E (IOT&E). It also describes conditions when it might be 
prudent to combine DT and OT, and finally identifies the 
T&E requirements for ACAT I and ACAT II programs. 

DAU maintains a curriculum to ensure T&E professionals 
are given the latest acquisition information. Career field 
developmental guides are available for each acquisition 
field, and break down paths for achieving certification 
within each respective acquisition profession, including 
training, education, and on-the-job experience. Similar to 
other acquisition career fields, T&E has three levels of 
proficiency, with suggested competencies for each. 

COMOPTEVFOR OTDG lists a variety of helpful resources, 
including the Test and Evaluation Community Network 
(TECNET, 2003, n.p.), which is stated to include virtually 
every testing resource the OTD will need, including 
resources from the other U.S. military services or from 
civilian services, either nationally or internationally. 

One of the responsibilities of the Deputy Director, 
Developmental Test and Evaluation OUSD (AT&L) is to ensure 
education and training of the T&E workforce, promote test & 
evaluation best practices, and to apply commercial 
practices to DOD programs. 

The Defense Test and Evaluation Professional Institute 
(DTEPI) serves as the executive secretary to the T&E 
Functional Board for the Defense Acquisition Workforce 
Improvement Act (DAWIA). The Director, DTEPI also chairs 
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both the Focus Group and the Competency Working Group. The 
Focus Group is composed of T&E experts from across the 
career field who develop competencies, which are the basis 
of the Defense Acquisition University (DAU) courses that 
are required for DAWIA certification for the T&E functional 
component of the Acquisition Workforce. The Competency 
Working Group reviews the competencies developed by the 
Focus Group and assigns a learning level to each task. 

DTEPI is chartered by the DOD Office of the Director, 
Operational Test and Evaluation (DOT&E) . The primary 
purpose of the Institute is to provide career and 
professional development, education, training, and 
recognition for the T&E professionals supporting the DOD. 
The Institute also is to serve as a forum for enhancement 
of the test and evaluation process to meet current and 
future needs and challenges. 

As part of the National Defense Authorization Act of 
2003 creating the Defense Test Resource Management Center, 
section 234, the Under Secretary of Defense for 
Acquisition, Technology, and Logistics was requested to 
submit to Congress a report on the capabilities of the test 
and evaluation workforce of the Department of Defense. 
Working with the Under Secretary of Defense for Personnel 
and Readiness and the Director of Operational Test and 
Evaluation, the following was specified as requirements for 
a comprehensive plan: 

1) The report shall contain a plan for 
taking the actions necessary to ensure that the 
test and evaluation workforce of the Department 
of Defense is of sufficient size and has the 
expertise necessary to timely and accurately 
identify issues of military suitability and 
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effectiveness of Department of Defense systems 
through testing of the systems. 

2) The plan shall set forth objectives for 
the size, composition, and qualifications of the 
workforce, and shall specify the actions 
(including recruitment, retention, and training) 
and milestones for achieving the objectives. 

The report needed to also include: 

1) An assessment of the changing size and 
demographics of the test and evaluation 
workforce, including the impact of anticipated 
retirements among the most experienced personnel 
over the period of five fiscal years beginning 
with fiscal year 2003, together with a discussion 
of the management actions necessary to address 
the changes. 

2) An assessment of the anticipated 
workloads and responsibilities of the test and 
evaluation workforce over the period of ten 
fiscal years beginning with fiscal year 2003, 
together with the number and qualifications of 
military and civilian personnel necessary to 
carry out such workloads and responsibilities. 

3) The Under Secretary's specific plans 
for using the demonstration authority provided in 
section 4308 of the National Defense 
Authorization Act for Fiscal Year 1996 (Public 
Law 104-106; 10 U.S.C. 1701 note) and other 
special personnel management authorities of the 
Under Secretary to attract and retain qualified 
personnel in the test and evaluation workforce. 

4) Any recommended legislation or 
additional special authority that the Under 
Secretary considers appropriate for facilitating 
the recruitment and retention of qualified 
personnel for the test and evaluation workforce. 

5) Any other matters that are relevant to 
the capabilities of the test and evaluation 
workforce. 
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The OUSD (AT&L) response to this request was a report 


to Congress entitled, "Capabilities of the Test and 
Evaluation Workforce of the Department of Defense." 


Component 

OTAs 

MRTFB 

Other 

Total 

i Army 





Military 

494 

41 

73 

608 

Civilian 

853 

— 

2,926 

848 

4,627 

Total 

1,347 

2,967 

921 

5,235 

Navy/USMC 





Military 

272 

1,310 

30 

1,612 

Civilian 

97 

1,915 

1,801 

■SQ 

Total 

369 

3,225 

1.831 


Air Force 





Military 

539 

3,404 

298 

4.241 

Civilian 

200 

3,647 

93 

3,940 

Total 

739 

7,051 

391 

8,181 

Defense Agency 





Military 

5 

81 

29 

115 

Civilian 

16 

144 

44 

204 

Total 

21 

225 

73 

319 

i Total, All Components 





Military 

HOElS 

4,836 

430 

jKE2! 

Civilian 

1,166 

8,632 

2,786 


Total 

2,476 

13.468 

3,216 

19,160 


Table 2. DOD T&E Workforce by Component (2002) 


F. T&E BEST PRACTICES 

In a 2001 study sponsored by the Deputy Director, 
DT&E, OUSD (AT&L), conducted under contract by Science 
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Applications International Corporation (SIAC), a series of 
companies were visited to determine a set of commercial 
industry test and evaluation "Best Practices" that may have 
DOD test and evaluation organizational and process 
applicability. These best practices were grouped under two 
categories. Category I was defined as best practices that 
are either easily implemented or have already been 
partially implemented. Category II best practices were 
those less easy to implement and requiring examination by 
stakeholder teams to determine feasibility and to develop 
structure and schedules. The findings from this study 
follow. Starting with Category I: 

Philosophy, Policy, Approach 

1. Recognize that testing is a way to identify 
and solve problems early in the process in 
order to control time, cost, and schedule 
late in the process. 

2. Recognize that best practices generate 
success and vice versa. 

Test Investment 

• Ensure early determination of the investment 
costs to acquire new capability for program 
support. 

• Require analytically sound ROI analysis for 
test investments. 

• Ensure cohesive (year-to-year) investment 
plans. 

Test Execution 

• Involve testers and evaluators very early: 

o Ensures testers know test requirements 

o Ensures developers know requirements 

for test 

• Capture test costs at program initiation. 

• Emphasize concurrent and integrated T&E. 
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• Institute formal quality check processes. 

• Use System Integration Laboratories and 
embedded instrumentation. 

• Give proper consideration to the use of 
external test capability in test planning. 

• Ensure testers control test planning, 
equipment, facilities, instrumentation, and 
test resources. 

• Continue to increase the use of modeling and 
simulation to expand the test process. 

Test Evaluation 

• Continue to increase the use of modeling and 
simulation to expand the evaluation context 
based on verified test data. 

Category II: 

Philosophy, Policy, Approach 


• Stabilize corporate leadership and test staff. 

• Focus on the quality of product and process to drive 
the efficiency and effectiveness of T&E. 

• Develop consistent processes to ensure consistent 
products. Incorporate T&E as a process enabler. 

• Increase T&E to assure product quality rather than 
reduce it to save T&E cost. 

• Use metrics and quality control processes to 
understand how well the test process is operating. 

• Implement efficient and effective test processes in 
order to compete. Keys: 

o Ensure T&E is consistently part of the decision, 
planning, and execution process. 

o Early commitment by all stakeholders on required 
T&E resources. 

o Certification of T&E processes and organizations 
(-ISO 9000) 

o Ensuring capital capability. 

Test Investment 
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• Charge cost of test investment back to the program. 

Test Execution 

• Charge full cost of testing to the program. 

• Emphasize multi-use T&E platforms. 

• Do not generally support the outsourcing of testing 
and evaluation. 

• Frequently use the Six Sigma or similar quality 
processes. 

• Automate data collection and archiving. 

• Benchmark in-house and within industry. 

• Use measurements and metrics. 

• Initiate programs to seek ten-fold reductions in the 
number of software tests required. 

• Integrate Master Test Plans and test execution with 
program resources and milestones. 

• Establish measures of effectiveness 

• Quantify risk for management decision when considering 
reduced testing. 

• Train the in-house test workforce in test engineering 
disciplines. 

Test Evaluation 


• Use Physics of Failure as a tool to predict and 

analyze system performance and shortfalls. 

• Correlate faults and solutions in a closed loop 

process to ensure problems are resolved. 

Test Philosophy/Process/Evaluation 

• Establish corporate internal web based sites for 

exchange of ideas, benchmarks, data, applications, and 
processes. Address data collection retrieval, 

archiving, modeling and simulation, and test and 

evaluation methods. 
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The recommendations from this best practices study, as 
presented at the International Test and Evaluation 
Association in November of 2001 were: 

• Implement or reinforce the Category I Best Practices 
in DOD as soon as possible. 

• Develop implementation or reinforcement strategies for 
Category II Best Practices using DOD T&E stakeholder 
teams. 

• Present the results of this study to the DOD 
acquisition and T&E communities. 
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III. OPEN SYSTEMS AND T&E 


A. OPEN SYSTEMS VERSUS OPEN SOURCE 

An Open System (OS) is a system that implements 
sufficient open specifications for interfaces, services, 
and supporting formats to enable properly engineered 
components to be utilized across a wide range of systems 
with minimal changes, to interoperate with other components 
on local and remote systems, and to interact with users in 
a style that facilitates portability. An OS is 
characterized by the following: 

• Well-defined, widely used, non-proprietary 
interfaces/protocols. 

• Use of standards which are developed/adopted 
by industrially recognized standards bodies. 

• Definition of all aspects of system 

interfaces to facilitate new or additional 
systems capabilities for a wide range of 
applications. 

• Explicit provision for expansion or 

upgrading through the incorporation of 
additional or higher performance elements 
with minimal impact on the system. 

(IEEE POSIX 1003.0/D15 as modified by the Tri-Service Open 
Systems Architecture Working Group, 2002.) 

The open systems emphasis in improved interfaces and 
interoperability provides opportunity for superior 
performance, accelerated delivery, and more affordable 
systems. Open systems is an "enabler" for a number of 
acquisition reform initiatives such as cost as an 
independent variable, performance specifications, use of 
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commercial items, and configuration management (Open 
Systems Definition, 2003 51.) 

B. OPEN SYSTEMS AND STANDARDS 


An open system design offers benefits such as life 
cycle support, affordability, and allowing timely 
technology insertion. However, there are substantial 
differences in the way open systems will have to be tested 
and evaluated. Whereas open system designs will rely on an 
increased use of commercial and non-developmental items in 
systems architectures, T&E will have to plan for 
significant technical differences. These differences will 
involve many aspects of engineering and management such as 
(DOD Open Systems Joint Task Force, 2003, 52) : 


• Standards-based architectures lessen the 
degree of control that DOD can expect to 
exert. Changes, fixes, and updates will 
likely be under the vendor's control, but 
adherence to the standard can be expected. 

• Standards-based elements of the architecture 
may be cheaper and faster to acquire but 
will not necessarily be cheaper and faster 
to integrate, update, test, and evaluate. 

• Selection may be risky. Open systems 

acquisition will demand that the program 
manager know substantially more about 
technology and the associated conditions of 
various vendors. 


• Standards 

evolve with time. While 

it 

is a 

challenge 

to 

visualize 

whether 

a 

given 

standard 

will 

endure. 

it may 

be 

more 

challenging to 

determine 

when to 

swap 

from 


one standard to another. 

• Integration becomes more important than 
design. Performance requirements must be 
realized without explicit control of the 
component design specification. 
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• Once integrated, a component may impact 
global system parameters. Testing will 
become an on-going and continuing activity 
to verify that COTS and NDI items can be 
successfully integrated into future systems. 

The following is defined in Volume 1.0 of the Navy's 
"DESIGN GUIDANCE FOR THE NAVY OPEN ARCHITECTURE COMPUTING 
CAPABILITY"(Strei, 2003, 51.3.3) 

An open system approach has become an important aspect 
of system design and development in a wide variety of 
enterprises. This is true primarily because open systems 
convey certain benefits in terms of reduced life-cycle 
cost, reduced time-to-market, increased ability to inter¬ 
operate and cooperate with others, reduced personnel 
training, etc. A number of open systems definitions exist 
within the literature. This guidance document adopts the 
definition developed by the DOD Open Systems Joint Task 
Force (OSJTF), which operates at the level of the Office of 
the Secretary of Defense: 

Open system: "A system that implements sufficient open 
specifications for interfaces, services, and supporting 
formats to enable properly engineered components to be 
utilized across a wide range of systems with minimal 
changes, to interoperate with other components on local and 
remote systems, and to interact with users in a style that 
facilitates portability." (DOD Open Systems Joint Task 
Force, 2003, 52): 

Open systems - and architectures built to open system 
principles - possess a number of common characteristics. 
While not every open system possesses every possible 
characteristic, most open systems tend to possess most of 
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these characteristics. Based on examination of the various 
open system definitions, the attributes of an open system 
include the following: 

• Use of public, consensus-based standards 

• Adoption of standard interfaces 

• Adoption of standard services (defined functions) 

• Use of product types supported by multiple vendors 

• Selection of stable vendor with broad customer base 
and large market share 

• Interoperability, with minimal integration 

• Ease of scalability and upgradability 

• Portability of application(s) 

• Portability of users 

C. NAVY OPEN ARCHITECTURE 

Navy Open Architecture (NOA) is an initiative to 
design and build a combat system that meets changing 
requirements into the 21st century, while also being 
rapidly and affordably upgraded throughout its life cycle 
(The Open Group, 2003). NOA plans to adapt and exploit new 
developments in open system design principles and system 
architectures, as well as standards-based computing 
technologies from the Commercial Off-The-Shelf (COTS) 
marketplace. 

The NOA Working Group plans to first develop a 
coordinated open architecture for real-time and embedded 
system environments that would be mutually beneficial for 
various architecture approaches to include but not limited 
to: DOD Joint Integrated Open Architectures, Navy Open 

Architecture, Air Force Viable Combat Aircraft Joint 
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Council on Aging Avionics, Modular Open Systems Approach 
Interoperability Initiative, Army Weapon Systems Common 
Operating Environment and various open architectures from 
corporations and system integrators. 


PROCESSING SYSTEM ARCHITECTURE EVOLUTION 

COMPUTER ARCHITECTURE EVOLUTION 
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Figure 12. Computer Processing Architecture 

Evolution 

This approach would leverage the information 
technology industry investment in the development of COTS 
components. The use of COTS should allow for easy 
transition to commercially available advanced hardware and 
software technology. The key to this approach however, is 
the use of COTS products that already use open standards. 
Open standards promote conformance at the interface level 
ensuring compatibility and interoperability. Development 
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of a fully open architecture would allow for the use of 
future technology and transition the insertion of 
components from one generation to the next based on 
hardware and software products that are conformant to open 
standards (Chief Information Office, The Open Group, 2003.) 

The Navy gradually wants to do away with decades-old 
proprietary combat-system software and replace it with a 
modern open architecture (Erwin, 2003, 11.) The cost to 

move MILSPEC, Navy-specific systems into commercial 
computing environments is difficult to calculate, but could 
save billions of dollars, over time. And an open 
architecture could help the Navy improve the capabilities 
of the AEGIS combat system for future missile-defense 
missions. 


An open architecture is what technologists 
call a "plug and play" computing environment, one 
that allows for easy upgrades of software 
applications, without having to reengineer a 
warship's entire combat system. "The analogy is 
that when you get a new refrigerator, you don't 
need to worry about testing the sink and 
everything else," said one industry expert 
(Erwin, 2003, 12.) 

Whereas the Navy already spends billions of dollars 
annually upgrading proprietary software, an open 
architecture is seen as the path to significant savings. 

There is undoubtedly a technological 
incentive surrounding open architecture. While 
pushing forward in such pursuits, the Navy must 
simultaneously provide new combat-system 
computers, while keeping the AEGIS fleet ready 
for war and meeting its missile-defense 
commitments. By 2005, the Navy is expected to 
deploy 18 anti-ballistic missile AEGIS warships— 
three cruisers and 15 destroyers (Erwin, 2003, 

13. ) 
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In charge of developing a plan to introduce open- 
architecture computers in the Navy by 2010 is a new 
organization created last year, the Program Executive 
Office for Integrated Warfare Systems (PEO IWS). PEO IWS is 
working with large and small companies to develop standards 
and protocols that will eventually influence every computer 
system in the Navy. 

Open architecture is "the right way to go" 
for the Navy, said Rear Adm. C. Tom Bush, the 
head of PEO IWS. "We need to stop building 
proprietary architectures." Bush believes that an 
open architecture can help make those upgrades 
easier and less costly, saving the Navy at least 
$1 billion a year (about 50 percent of the 

service's annual expenditures on software 
upgrades) . "On a good day, when something needs 
an upgrade, it requires seven to 28 changes," 

Bush said. "We can't build a combat system for 
every ship. But we can build a single 
architecture." (Erwin, 2003, 55-7.) 

Another benefit of open architecture is 

"interoperability," said Rear Adm. Henry G. 
Ulrich III, director of Naval Surface Warfare. 

"The business model and the architecture we have 
now are driving us away from interoperability, 
not only with our allies but amongst ourselves. 

... The current technology, which is not 

compatible with anything else, drives up the cost 
of producing and upgrading software." 

The Navy's director of open architecture is 
Tom Pendergraft, a career combat-system engineer. 

Upon taking the job only a few months ago, he 
quickly learned that bringing open architecture 
to the Navy is less about technology and 

engineering than about "culture and business 
models changing," he said in an interview. The 
enormous expense associated with software 

upgrades and a desire to improve the current 
technology make it imperative for the Navy to 
begin migrating to open architecture, he said. 
Upgrading the AEGIS combat system on average can 
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cost hundreds of millions of dollars. Not only 
can the service not afford these prices, but in 
many cases, the computers have been upgraded so 
much already that their capacity to grow has been 
exhausted. "Necessity is the mother of 
invention," Pendergraft said. "That is what we 
are talking about here." 

At the center of the Navy's theater defense capability 
is the AEGIS Weapon System. With a phased-array radar that 
can track hundreds of targets simultaneously, and a command 
and control computer system allowing simultaneous tracking 
operations in air, surface and undersea warfare, keeping 
pace with the emerging threat is critical. Detect, track 
and engage functions require enormous computing capability, 
with millions of actions being performed by the host weapon 
system every second. 

In the early days of AEGIS, said 
Pendergraft, "we had a single computer that did 
all the computation for warfare systems." As the 
operations became more complex, when the Navy 
started using more advanced weapons, the 
computing power needs grew exponentially. 
Another drawback to the current technology is 
that it is "serial," meaning it can do one thing 
at a time - detection, tracking, identification, 
decision, engagement. "You only had one computer 
to do all that. ... Our architecture is serially 
based, with point-to-point connectivity," said 
Pendergraft. "Pretty soon, you have what we call 
spaghetti code." (Erwin, 2003, 19.) 

This "spaghetti code" is commonly the reason why 
upgrades are so expensive. Besides the fact that this 
legacy code can only be maintained by a shrinking resource 
pool, when a single applications is upgraded, whether to 
fix a software bug, or to build in more capability, the 
entire weapon system has to be tested to make sure no 
changes were made inadvertently to other functions. 
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"Today, when we make a change, by the nature 
of the shared-memory architecture, you end up 
having to retouch the entire system," said 
Orlando Carvalho, vice president of AEGIS 
programs at Lockheed Martin Corp. "In some cases, 
you have to make many changes for a fairly small 
upgrade." Norm Malnak, Lockheed Martin technical 
director, said the problem is exacerbated by the 
presence of multiple AEGIS baselines (software 
releases) throughout the fleet. The oldest ships, 
for example, use baseline 1.4. Others have 

baselines 2.1, 5.3 or 6.3, for the newest ships. 

Lockheed Martin is developing baseline 7.1, with 
more advanced features. An open architecture will 
help "get commonality across the fleet ," said 

Malnak. "That saves a lot of money." (Erwin, 

2003, 111 .) 

The Navy has spent hundreds of millions of 
dollars on modern computers to expand the memory 
and computer processing speed of AEGIS combat 

applications, but fast PCs is not what open 
architecture is about, explained Pendergraft. "We 
went to COTS computers, but we haven't done 

anything with all the point-to-point 

connectivity. . . . With open architecture, we are 
changing the fundamental structure." 

Navy Standard Computers are used to process the input 
and output data. AEGIS uses several variants of the 
AN/UYQ-43 computer, but each lacks the speed and memory of 
personal computers found commonly in the office or in the 
home. The Navy has increased computing power through the 
use of adjunct processors and additional memory, but the 
UYQ-43 handles the critical functionality that eventually 
builds fire control solutions, leading to ordnance on 
target. 


"Our current Navy Standard Computers are at 
about 99 percent capacity," said Pendergraft. 
"Every time we want to add a new function, we 
can't do it on NSC, so we add adjunct computers." 
This setup still maintains the "spaghetti code" 
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structure. By adding more processors and 
functions, "all the stuff starts crisscrossing. 

We have point-to-point spaghetti code all over 
the place. It makes it very complicated and 
expensive to maintain." Further, "we are 
prohibited right now from adding a lot of 
significant war-fighting capability, because we 
don't have the computing capacity," he said. 

To make the open architecture plan work, "we 
have to stop people from putting proprietary 
computers on ships. That is what kills us. Every 
time there is an upgrade or the manufacturer goes 
out of business, we are toast. We have to hire 
someone to rebuild that system, or we have to 
keep someone in business for a lot longer than we 
want to." Unfortunately, he said, "There is no 
police force for specs and standards." (Erwin, 

2003, 113 .) 

Open architecture requires a significant up-front 
investment and questions about its claimed merits. PEO has 
provided various estimates of what it would cost to convert 
the entire fleet, but the debate over legacy development 
versus open systems development seems to be winding down. 

D. OPEN SYSTEMS AND MDA 

As the Navy moves closer to fielding a ballistic- 
missile defense for the United States, the ability of 
computers to do the job comes into question. Even in the 
most modern of deployed weapon systems, the computing 
environment is taxed to the point that further enhancement 
or upgrades may not be possible. According to PEO IWS's OA 
lead, Tom Pendergraft, "current computing plants are pretty 
full. If you want to add BMD on top of that, that is going 
to be pretty tough. ... If we go to open architecture, with 
distributed computing, we would have virtually unlimited 
[computational power] resources." (Erwin, 2003, SI 17. ) 
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It is possible to accomplish missile-defense 
missions in legacy AEGIS ships, "assuming some 
modifications to open up the architecture," he 
said. "You are not going to get there with one 
computer ." 

Future missile-defense capabilities the 
Pentagon envisions, such as new solid-state radar 
and extended range weapons, will require more 
computing power than currently exists. "As you 
move forward with missile defense, you want 
additional signal processing capability ," said 
Chris Myers, director of missile defense and 
radar programs at Lockheed Martin. The company is 
responsible for various pieces of the naval 
missile-defense program, including AEGIS, cruiser 
upgrades, and the development of an active solid- 
state radar. 

We want to upgrade those computing plants so 
it makes it a lot easier to upgrade AEGIS," said 
Myers. "In the future, you want to see targets 
further away, smaller things, you need additional 
radar power and sensitivity to see that. ... 

There is additional computing power required as 
you move to a solid-state radar. (Erwin, 2003, 

519. ) 

Lockheed is one of 49 companies that 
received contracts to help the Navy come up with 
commercial standards and protocols for the open 
architecture. The plan is to begin installing the 
new technology on surface ships and then expand 
to submarines. The DDX land-attack destroyer, to 
be deployed by 2012, is expected to be the Navy's 
first truly "open-systems" ship. 

In March, PEO IWS released an interim set of 
specifications and standards that new programs will have to 
follow, in order to be open-architecture compliant. 
Existing legacy systems however, will have to be addressed 
as well, but there are challenges. Existing weapons systems 
using dated technology require very specific techniques and 
talents to upgrade, which will likely be very expensive. In 
addition, DON continues to train and fight wars with ships 
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that cannot afford to be taken out of service for the time 
it would take to complete a comprehensive upgrade, let 
alone test and evaluate. Moving into OA standards allows 
future systems to evolve, but will also allow existing 
systems to eventually become more open. 

A transition to open architecture, for 
example, would involve "taking pieces of our 
combat systems, throwing away the old code, rack 
and stack the algorithms, write them in modern 
computing program so they can run on a modern 
computing environment," said Pendergraft. "In the 
AEGIS program, we are starting now to open up the 
system," he said. 

Rick Scharadin, program manager at Lockheed 
Martin, said the company will demonstrate how 
segments of AEGIS can be converted to open 
architecture in a piecemeal fashion. The first 
step is to upgrade the computing environment for 
the radar, by 2006. The second piece is to 

convert the displays, by 2007. In 2008, the plan 
is to demonstrate open-architecture radar and 
displays, weapons control and fire control. "The 
key is to find those parts that you could easily 
remove," said Pendergraft. "The only way we'11 be 
able to do this is one part at a time. Can't do 
it all at once." (Erwin, 2003, 513-21.) 

PEO IWS plans to spend about $50 million a 
year on research related to open architecture. A 
lot more money, however, will be needed to 

upgrade ships. Those funds may have to come from 
ongoing acquisition programs, a prospect that 
Pendergraft acknowledged will stir the proverbial 
"rice bowls" associated with military projects. 

"Some of the programs of record are going to have 
to change direction in order to pay for this," 
said Pendergraft. 

"In a front-line combat system like AEGIS, 
you cannot do plug and play without doing 

specific reengineering to make sure you haven't 
contaminated the system integrity," said retired 
Rear Adm. George Meinig, who was the AEGIS 

technical director in the mid-1980s. "The 
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benefits of open architecture are still 

desirable," he said. "But there is no assurance 

that unaltered commercial products can meet the 
performance requirements of the combat system. 

. . . You have to do careful testing to make sure 
the design is compliant with the requirement and 
that you haven't messed up the whole system." 
(Erwin, 2003, 525.) 

As the benefits from employing open architectures are 
becoming more well understood, the return on investment 

might not be seen for many years. And the initial 

conversion to an OA would not necessarily increase the 
capability right away, but instead allow for the potential 
growth in the future. So as current, in-service weapon 


systems are on the verge of obsolescence, open systems 
architecture could, in concept, give them the ability to 
serve the warfighter into the future. 


According to some very recent educational forums 
sponsored by DAU and focused on Program Management looking 
towards the future, open architecture can be summarized 
with the following aspects: 


• Today's Fleet computing architectures are performance 
limited and expensive to upgrade. 

• Implementation of warfighting functions using standards- 
based solutions will enable common, interoperable 
capabilities to be fielded faster at reduced cost. 

• Rapid Technology Insertion Program (RTIP) will provide a 
structured approach for introduction of OA components 
into the Fleet (Program Managers Workshop, 2003.) 


E. NOA - EMERGING GUIDANCE 


From Version 1.0 of DESIGN GUIDANCE FOR THE NAVY OPEN 
ARCHITECTURE COMPUTING CAPABILITY, Navy open architecture 
is the high-level technical structure of the weapon system 
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as designed in accordance with the principles of open 
systems to achieve both real-time mission requirements and 
life-cycle supportability goals. Technical characteristics 
of NOA include: 

• Distribution of processing 

• Widespread use of standards-based COTS computing 
technologies 

• Functional capabilities implemented as medium- 
grain components 

• Use of object oriented (00) programming within 
components and middleware technologies for 
interconnection of and interoperation among 
components 

• Use of design mechanisms such as client-server to 
maximize isolation of implementation details from 
publicly visible services and APIs 

• Portability and transparency of application 
components with respect to physical location and 
network, processor and operating system types, 
etc. 

The corresponding goals of the NOA are to provide to 
the weapon system not only the benefits of assured 
technical performance, but also of reduced life-cycle cost, 
affordable technology refresh, and reduced upgrade cycle 
time. Expected benefits include: 

• Scalable, load invariant performance 

• Enhanced information access and interoperability 

• Enhanced system flexibility for accomplishment of 
mission and operational objectives 

• Enhanced survivability and availability 
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• Reduced life-cycle cost and affordable COTS 
technology refresh 

• Reduced cycle time for changes and upgrades. 


Navy OA Functional Architecture...Proposed End State View 
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Figure 13. Notional NOA Functional 
Architecture 


F. NOA GOALS AND FUNCTIONAL ARCHITECTURE 

In broad and general terms (Strei, 2002), architecture 
is defined as "the structural design of an entity." Adding 
"openness" to the list of architectural characteristics 
implies that the "structure" of the architecture explicitly 
promotes interoperability, both internally and externally, 
as well as ease of modification and extension. 
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It is an engineering truism that what is 
achievable in system design (architecture) is a 
function of not only the task to be accomplished 
but also the technologies that are available. 
However, the evolution of high performance COTS, 
combined with continued growth of weapon system 
and combat system requirements, provides an 
opportunity to design an architecture more 
capable of exploiting new technologies than the 
federated legacy architecture that has served the 
Navy for well over two decades. The need for 
evolution toward an open architecture is 
motivated by both performance and supportability 
considerations. Commensurate with this dual set 
of motivating factors, the goals of the NOA are 
as follows: 

• Combat system, weapon system, command support 
system and HM&E capabilities that continue to 
pace the threat. 

• System design that fosters affordable 
development and life-cycle maintenance. 

• System design that reduces upgrade cycle time 
and time-to-deployment for new features. 

• Architecture that is technology refreshable 
despite rapid COTS obsolescence. 

• Improvements in NWS Human Systems Integration. 

Finally, system requirements may include not 
only capability and performance goals but also 

non-functional engineering goals as well ("- 
ilities"). In addition to traditional metrics 
such as reliability and survivability, NOA 

metrics include qualitative goals such as 
portability, scalability, extensibility, and 

flexibility of use. These goals will be met, in 
part, by careful design and in part through use 
of open systems principles and standards. This 
document focuses primarily on the technical 
aspects of designing NOA. However, in many 

cases, the recommended design choices and 

technologies are chosen with the goal of 
supporting this qualitative metrics as well. 
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G. WHY OPEN ARCHITECTURE? 

In a recent Defense News article discussing the U.S. 
Navy's decision to change its acquisition strategy for CEC, 
replacing a winner-take-all approach with a series of 
smaller competitions, the rationale was OA. 

"We're totally changing the plans for CEC Block 
2 ," said a Navy official. Instead of a closed 
system. Block 2 will incorporate open- 
architecture standards to hold costs down, allow 
more joint interactions, and help the fleet to 
adapt more quickly to new threats. Service 
officials hope this new approach will encourage 
innovation, entice more firms to compete for the 
work, and ultimately push down purchase costs 
(Sherman, p.l, 2003.) 

Open architecture is the key to affordable 21st 
Century joint combat capability (Program Managers Workshop, 
2003.) OA enables current weapon systems, and 
corresponding computing systems, which today cannot support 
emerging Sea Power 21 warfighting capability requirements, 
to be upgradeable to meet these future needs. In addition, 
OA is claimed to be affordable, although this is a subject 
of great debate at present. What is not debatable is that 
current, in-service computing architectures are 
unaffordable. Presently, each ship class addresses common 
problems uniquely, while software and hardware changes are 
interdependent. Finally, OA must support Joint 
Interoperability. Today's existing in-service 
architectures cannot support Forcenet implementation. 

The OA implementation strategy must include several 
concepts. To begin with, computer program upgrades that 
provide only marginal warfighting capability enhancement 
must be frozen and future upgrade plans terminated. OA 
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technical and functional architectures must be completed 
and consensus gained for scaleable. Navy-wide applications. 
A Rapid Technology Insertion Program process has been 
proposed to transition promising technologies to certified 
warfighting products. All new systems must comply with 
agreed-upon and documented OA standards specifications and 
guidance. Finally, proponents of open architecture should 
pursue coordination and agreements with all other 
initiatives and programs. 
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IV. T&E IN THE FUTURE 


A. INDICATORS 

In an Inside the Navy article dated September 8 2003, 
entitled, "Study prepared for Young: Cohen predicts Navy 
will put EM gun on DD (X) within a decade," The navy plans 
to have an electromagnetic rail gun aboard a DD (X) 
destroyer in eight to 10 years, according to Chief of Naval 
Research Rear Admiral Jay Cohen, who expects the total cost 
of developing one or two gun prototypes would be $500 
million to $1 billion. Though not explosive, EM gun 
projectiles would hit targets with uncanny speed and 
devastating force, setting a new standard for deadly, long- 
range shipboard guns. "I will tell you, we think in eight 
to 10 years, you're going to see a 250-to-300 mile 
electromagnetic rail gun on DD(X)," Cohen predicted in a 
presentation at "COMDEF 2003" in Washington, DC. 

"We must acknowledge that our way of war requires superiority in all 
mediums of conflict, including space. Thus, we must plan for and 
execute to win space superiority." 

Gen. Richard B. Myer, Vice-Chairman, U.S. Joint Chiefs of Staff 

"The idea of putting weapons in space to dominate the globe is 
simply not compatable with who we are and what we represent as 
Americans.” 


Figure 14. From NMD: The Arctic Dimension 

(2003) 

While there is plenty of speculation and high hopes 
for future systems to be developed and acquired in the 
future, but there is very little evidence of how we might 
prepare to test and evaluate these emerging systems. In 
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addition, there appears to be no consensus about what 
testing in the future will look like. 

B. HOW DOES OPEN SYSTEMS ARCHITECTURE CHANGE THINGS? 

In a speech by RADM Phillip M. Balisle, Director, 
Surface Warfare Division (N76) to the Surface Navy 

Association (March 2002) he stated, "As we explore the 
transformation of the existing AEGIS baselines into an open 
architecture, distributed processing combat system, we 
intend to build these interoperability enhancements into 
our new systems from the ground up." He continued by 

saying, "Following the successful transition to a complete 
COTS computing environment on our new construction AEGIS 

DDGs, AEGIS baseline development will introduce an open 
architecture, high performance, interoperable and network 
ready software architecture, which will eliminate many of 
the interoperability limitations of today's combat 
systems." He concluded his speech by stating, "When we 
align our systems and integrate them using a systems 
engineering approach into a new architecture which allows 
for the efficient exchange of required data across the 
network, we will realize another dramatic increase in 

situational awareness, speed of command and synchronization 
that will buy back even more critical battle space for our 
Warfighters . " 

C. T&E IN THE MISSILE DEFENSE AGENCY 

The Missile Defense Agency, in July 2003, released the 
following information regarding test and evaluation (BMD 
Test & Evaluation, 2003, pp. 1-2.): 

The Ballistic Missile Defense (BMD) test 

philosophy recognizes the need for an integrated. 
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phased test program that comprehensively covers 
all facets of testing. Testing components, 
subsystems and systems, especially early in the 
developmental cycle, can determine current 
performance capabilities and identify potential 
design areas where technology can increase 
overall system capability. Later testing 
demonstrates and measures the effectiveness and 
suitability of missile defense systems in their 
intended operational environments. 

The BMD System (BMDS) test methodology adds 
system complexities over time. For example, 
system performance in the presence of 
countermeasures and operations in increasingly 
stressful combat scenarios would be addressed in 
segmental tests. This step-by-step approach 
facilitates timely assessments of the most 
critical design risk areas. 

The MDA test and assessment program supports 
credible decisions with respect to the BMDS and 
its elements. Specific program objectives focus 
on: characterizing, demonstrating, measuring and 
verifying achievement of BMDS capabilities; 
executing BMDS test events; facilitating credible 
testing of BMDS Elements, technology experiments 
and international collaborative programs; and 
anchoring Modeling and Simulation with test data 
for use in measuring performance throughout the 
test envelope. 

Meeting the challenges of BMD testing 
requires an extensive test infrastructure. 
Collectively, groundtest facilities, ranges, 
sensors and instrumentation assets provide 
valuable BMD program-wide risk reduction and test 
capability to assess BMD system and element 
performance. Ground tests are conducted at high¬ 
speed sled tracks, hardware-in-the-loop 
facilities, aero-ballistic ranges, aero-optic and 
aero-thermal shock tunnels and space chambers. 

MDA deploys mobile airborne sensors to 
ranges during flight tests, which have onboard 
signal and data processing and collection 
capabilities. More recently this includes the 
development of transportable instrumentation and 
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common standards to support MDA testing with 
flexible scenarios at a variety of locations. 

MDA conducts BMDS Integrated Tests using 
selected hardware and software from the 
individual elements to investigate performance, 
joint operations and interoperability. These 
tests include the Critical Measurements Programs 
and the System Integration Tests. The former are 
live test flights that provide common data 
collection opportunities and the latter are live 
intercept tests involving representations of 
potential future threats. Results of all tests 
are used to conduct annual system- wide 
capability assessments. 

D. THE FATE OF OT - ONE EXAMPLE: MDA 

The Bush administration is proposing to exempt the 
Pentagon's controversial missile defense system from 
operational testing legally required of every new weapons 
system in order to deploy it by 2004 (Schrader, 2003, p.l.) 

In the FY 2004 budget, is a request to 
rewrite a law designed to prevent the production 
and fielding of weapons systems that don't work. 

If the provision is enacted, it would be the 
first time a major weapons system was formally 
exempted from the testing requirement. The 
proposal follows administration moves to bypass 
congressional reporting and oversight 
requirements in order to accelerate development 
of a national missile defense system. One of 
Bush's goals when he took office was to carry out 
a missile defense system — an idea first proposed 
by President Reagan — and he almost immediately 
expanded the scope and the funding of the 
controversial program, which had encountered 
scientific and budgetary difficulties in recent 
years. 

Last year, to help achieve that goal. 
Defense Secretary Donald H. Rumsfeld gave the 
Missile Defense Agency unprecedented managerial 
autonomy and removed procurement procedures that 
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were intended to ensure new weapons programs 
remain on track and within budget (Schrader, 
2003, p.2.) 

While the exemptions granted previously gave 
the missile defense program an unprecedented 
degree of autonomy from congressional oversight, 
they did not exclude it from testing. 
Highlighting its technical weaknesses has been 
opponents' best hope for slowing the long-debated 
program. In recent years, critics repeatedly 
have used Pentagon data from missile defense 
flight tests to challenge whether the experiments 
were as successful as claimed. 

The latest proposal from the Pentagon would 
exempt the missile defense deployment from a law 
that requires the Defense Department to certify 
that appropriate operational testing has been 
completed before putting systems into production. 

The Bush administration announced in 
December 2002 a goal of having a limited ground- 
based system operational in Alaska and at 
Vandenberg Air Force Base in California by Oct. 
1, 2004 (Schrader, 2003, p.3.) "The moves last 

year were just about reporting requirements. 
This is different," said Philip Coyle, director 
of operational testing and evaluation for the 
Pentagon from 1994 to 2001. "This is about 
obeying the law. Without these tests, we may 
never know whether this system works or not, and 
if they are done after this system is deployed, 
we won't know until we've spent $70 billion on a 
ground-based missile defense system." 

In a letter to Rumsfeld, Feinstein wrote: "I 
believe that any deployed missile defense system 
must meet the same requirements and standards 
that we set for all other fully operational 
weapons systems. Indeed, given the potential 
cost of a failure of missile defense, I believe 
that, if anything, it should be required to meet 
more stringent test standards than normally 
required." 

Rumsfeld replied that an exemption made 
sense in the case of missile defense (Schrader, 
2003, p.4.) "I happen to think that thinking we 
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cannot deploy something . . . until you have 
everything perfect, every ' i' dotted and every 
' t' crossed, it's probably not a good idea," he 
said. "In the case of missile defense, I think 
we need to get something out there, in the 
ground, at sea, and in a way that we can test it, 
we can look at it, we can develop it, we can 
evolve it, and ... learn from the experimentation 
with it." 

Rumsfeld pointed out that two other weapon 
systems in recent years — the Predator unmanned 
aerial vehicle and the Joint-STAR aircraft radar 
systems — were deployed before they were tested 
operationally. But those systems did eventually 
go through operational testing, and neither went 
into full production until the testing was 
completed. There is no guarantee the operational 
testing will ever take place if the law is 
changed to allow the system to be deployed 
(Schrader, 2003, p.5.) 

E. WHERE MODERN T&E MUST EVOLVE 

To meet the challenges presented by an evolutionary 
acquisition, and a US Navy deep into transformation, 
requires a T&E methodology that is equally as 
transformational. 

The Navy is actively engaged in the acquisition of 
future, technology-exploiting weapon systems. In a recent 
article (Schweizer, 2003, p.5) discussing the merits of 

directed energy weapons, "Navy officials admit there's 
plenty of hard work ranging from basic science to rigorous 
operational evaluation - to be done before some of these 
systems sail aboard a warship. Even so, it's a generation 
of weapons that is tantalizingly close to becoming 
reality." The good news is that someone is giving some 
thought to the issue of evaluation, meaning that the notion 
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of test and evaluation is not lost in the active pursuit of 
a desirable new technology and corresponding weapon system 


which will certainly 

bend the 

envelop for 

testing at 

our 

present ranges. 

and 

using 

our present 

targets. 

In 

addition, during 

the 

month of 

October 2003 

, ITEA will 

be 


conducting a symposium on understanding direct energy, with 
implications towards T&E. Perhaps the T&E community of 
practice is already on the right path. 

F. AEGIS T&E - LOOKING FORWARD 

The AEGIS T&E Process has always been intended to be a 
universal process, with applicability to both government 
and contractor personnel in all phases of the acquisition 
cycle, developmental, operational, or combined. Its use 
implements a "Build a little Test a little" philosophy and 
stresses testing before expending ordinance. 

Discipline in this test process is recognized as a 
contributor to cost effective system acquisitions that 
satisfy the Navy's needs. A disciplined and well-structured 
test program reduces the risk of acquiring an ineffective 
system and provides the program manager with timely 
information required to make prudent decisions during 
system development. 

Testing ranges from early component testing at the 
factory, to full system live fire performance 
demonstrations in a simulated real world environment. 
Regardless of the type of test, there are five guiding 
principles to help ensure the system under test fulfills 
its intended purpose. 
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1. Develop meaningful and applicable test 
objectives, and adhere to them in an orderly, repeatable, 
and disciplined manner. 

2. Use the closed loop systems engineering approach, 
from concept, to component, to subassembly, to subsystem, 
to system, to whole ship test. 

3. Test as early as possible and as often as 
affordable to find and correct problems before they become 
too costly. 

4. Involve the user, developmental tester and 
operational tester in the initial formation of the systems 
engineering council to develop test objectives to ensure 
continuous and timely information exchange of objectives 
and test results. 

5. Take the time to ensure all parties (developer, 
contractor, and government operational testers) thoroughly 
understand the systems mission requirements and agree on 
how the system will be tested, scored and evaluated. 

The need to take a disciplined approach in AEGIS T&E 
has been demonstrated many times in the past. Risks must 
be understood and controlled. Once a latent deficiency 
manifests itself, it is no longer a risk; it is a problem. 
The AEGIS Test and Evaluation community is an essential 
means of identifying, understanding, addressing system 
issues within both hardware and software. 

To evolve in the future, AEGIS T&E will complete five 
objectives for each improvement, modification, or system 
mission change: 
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1. Verify that test results are credible and support 
system acquisition milestones for decision-making. 
Incorporation of an OA software implementation system 
performance should be the same if not better then the 
previous legacy system. 

2. Provide early identification of AEGIS performance 
and supportability deficiencies for resolution. When 
limitations are discovered, they must be addressed as soon 
as possible to support further tests of performance. 

3. Identify and measure performance parameters that 
are critical to operational effectiveness and suitability 
through rigorous analysis and evaluation during the 
evolution of system requirements. 

4. Provide early identification and timely acquisition 
of test resources and assets necessary to stress the 
system. T&E assets are required to meet the approved test 
objectives and provide a means to verify specification 
compliance. 

5. Execute test programs that consistently apply the 
closed loop systems engineering approach to T&E. 
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V. CONCLUSION 


A. RECOMMENDATIONS 

The Navy has invested billions of dollars into weapon 
systems that are increasingly complex and are still 
evolving. Changes to computer program architecture and 
introduction of COTS equipment illustrates the Navy's 
requirement to improve existing systems and hulls. With 
this evolution T&E culture must also evolve to support 
future weapon systems that are increasingly complex and 
agile. It is not possible to proceed forward doing business 
the way it has been done in the past. Change in 
configuration forces the evolution of T&E. To evolve with 
the systems the T&E community must be vigilant in the 
following areas: 

Agility - To be adaptive to evolving threats, 
increasingly complex weapon systems, and more and more 
stressing operator training needs. The T&E Community must 
evolve with the systems and structure the evaluation 
performance in step with the newly imbedded technology. 

Flexible - Being able to address whatever new 
requirements are implied with the improvements or upgrades 
to the systems. 

Meaningful - Bringing to the event the regiment and 
expertise already being applied to legacy systems, 
validated test objectives and measures of effectiveness, 
suitability and performance. 

Repeatable - The T&E community must be able to sustain 
a benchmark for regression of each system. Core 
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capabilities may be improving, but each ships system was 
designed to support a specific mission. No matter what the 
change in computer program architecture or system hardware 
improvement, the ships system must fulfill its mission. 
Regression testing insures compliance, and repeatability. 
The T&E community must be capable of evaluation of 
performance to verify core functionality and the ability of 
the system to satisfy its mission. 

Innovation - The T&E community must find solutions for 
difficult scenarios blending a mix of live and modeled 
testing to gauge system performance, and to also provide a 
value-added operational feel for the warfighter. OA brings 
the promise of greatly enhanced and rapidly upgradeable 
systems, and with that promise comes the need for creative 
and innovative T&E solutions. 

Expertise/Lessons-Learned - The T&E professional 
workforce, who is the backbone for conducting modern-day 
T&E, must ensure that the collective knowledge for the 
business of test and evaluation is recorded, and passed on 
to the next generation T&E professionals. The AEGIS T&E 
community has evolved with a regiment and infrastructure 
based on lessons learned over the last 23 years of system 
test and certification. As the AEGIS program puts to sea 
its final ships and the AEGIS Weapon System reaches a final 
configuration, the AEGIS T&E community of practice must be 
preserved and applied to future, and evolving systems. 

Cost Effective - T&E must provide meaningful and 
measurable metrics, which demonstrate conclusively, the 
merit to T&E. The pitfalls and tradeoffs from inadequate 
testing must be readily available to help tomorrow's 
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decision makers to defend the level and appropriateness of 
T&E in the future. 

Technology Adopters - T&E must leverage technology 
wherever and whenever possible as a workforce multiplier, 
and a resource-saver. Just as future weapon systems 
embrace technology to provide new answers to difficult 
problems, so should future test and evaluation. Up front 
investments in technology are needed to ensure this 
happens. 

Safety - Weapon system test execution must remain 
safe. Weapon system complexity challenges the DOD's 
ability to design scenarios to adequately understand the 
performance-related aspects of systems undergoing test. 
Regardless of testing complexity, safety cannot ever be 
compromised. Pressure to reduce safety standards and 
practices to expedite programs, and thereby reduce costs, 
must be resisted. 

Environmental Compliance - Weapon system test 
execution must be in compliance with environmental laws and 
policies. At-sea testing is restrictive and difficult to 
characterize the impact to the environment. New future 
weapon technologies bring the challenge of additional 
review for environmental compliance. Increased weapon 
system complexity further challenges our understanding and 
ability to estimate impact upon the environment. The time 
and costs associated with adequate environmental review are 
prerequisite, and cannot be avoided. 

Test Where It Makes Sense - Current sea-based test 
ranges typically involve test areas instrumented for live- 
fire exercises out to 100 nautical miles offshore. 
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Increased weapons system lethality, range, and performance 
are features of most future navy weapon systems. To 
adequately test these systems at-sea without compromising 
safety and environmental policies, the testing is being 
pushed further offshore, away from traditional land-centric 
test range infrastructure. Because offshore waters are 
being encroached more and more by commercial and private 
boat traffic and air traffic, adequate test areas free from 
encroachment must be pushed further away from traditional 
land-based test ranges. Future testing will need to be 
conducted in open-ocean areas using both remote and 
autonomous test procedures/capabilities. Major development 
and investment in unmanned systems operations is needed to 
make possible open-ocean testing. Telemetry (data 
collection) systems will be particularly challenging. New 
open-ocean test areas might need 200-400 nautical miles of 
instrumented range. This requires both a major cultural 
change as well as increased T&E funding resources. 

Affordable - T&E processes and approaches can be cost- 
effective, yet still be unaffordable. The costs of testing 
current and future weapon technologies have been increasing 
for more than two decades. The ability to preserve costly 
special purpose test assets, test processes, and unique 
test range infrastructure is getting more difficult and 
challenging. T&E complexity is directly proportional to 
weapon system complexity. To remain affordable, T&E 
Programs are now expected to "get-it-right" the first time 
to keep costs affordable and manageable. But, T&E 
affordability cannot be measured in the test infrastructure 
costs, but rather, weapon system life-cycle costs. 
Affordable T&E should be measured by the degree total 



weapon system life-cycle costs are minimized and reduce 
associated risks. 

B. CONCLUSIONS 

Test and Evaluation of ships systems verify that the 
Navy gets what it paid for, and the systems perform the 
mission they were designed to do. Cost effectiveness and 
expedience in the face of evolving technology are the 
hallmarks of a good T&E community. The lives of those who 
operate and maintain today's weapon systems depend on solid 
and reliable testing, to ensure systems do what they were 
designed to do. The T&E community makes it possible. 

In a letter to the editors of Scientific American 
(Sawyer, 2003, p. 20) in response to an earlier article 
entitled, "Misguided Missile Shield", the writer states, "a 
demand for perfect realism in testing a complex weapon 
system like missile defense is unrealistic. More testing 
in necessary - more tests, however, are scheduled." Indeed 
it would seem that testing of as many of the variables 
possible is prudent, until the T&E community can respond 
with more comprehensive and full-scale tests in 
environments which mirror the conditions anticipated where 
these future weapon systems will be operated by tomorrow's 
warfighters. 


C. SUGGESTED TOPICS FOR FUTURE RESEARCH 

This research provided a historical account of several 
ongoing and emerging Navy T&E programs with the goal being 
to provide a series of attributes T&E must exhibit to 
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successfully field future systems. While touching on 
certain indicators and spending some focus on AEGIS Weapon 
System development and open architecture, the following 
topics are areas that should be considered for future 
research. 

• Analyze "lessons learned" from evolutionary 
acquisition to show how programs are balancing new 
capabilities and lifecycle support against T&E 
abilities and needs. T&E must evolve and transform to 
provide continuous test windows inside the systems 
development, as well as being as operational as 
possible. 

• Assess the progress of AEGIS Open Architecture, and 
show mapping against Navy Open Architecture. As 

standards are continually being developed and vetted 
out in the technical community, the real success lies 
in bringing actual open systems direct to the 
warfighter. Before this can happen however, these 
standards must be agreed upon and current and emerging 
systems will have to adopt them unilaterally. 

• Compare and contrast the decision to convert the AEGIS 
Fleet into an open systems baseline, versus bringing 
the AEGIS Weapon System into a "caretaker" status . At 

present, the future baseline configuration for both 
AEGIS Cruisers and AEGIS Destroyers is still a matter 
of great debate, and very much dependent on future 
budgetary decisions and an unstable political horizon. 

• Evaluate the challenges of providing effective joint 
and allied systems T&E. In addition, explore the need 
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for consistent interoperability standards, and how 
open systems development may help or hinder the 
interoperability crisis plaguing many major in-service 
weapons systems today. 

• Research a case example such as DD (X) as a "cradle to 
grave" open architecture program currently undergoing 
requirements generation and definition phase. Explore 
the techniques for building an open computing 
environment that will incorporate test and evaluation 
inside the actual tactical code. 


The 1970 Blue Ribbon Defense Panel, also know as the 
Fitzhugh Commission, took a very serious and in-depth 
review of defense acquisition policies and procedures. 
Their finding led to sweeping recommendations, which 
changed modern acquisition well before the term 
"evolutionary acquisition" was coined. This Commission 
also made profound recommendations concerning T&E, which 
actually led to the establishment of both the office 
overseeing DT&E as well as OT&E. In the thirty plus years 
since the Fitzhugh Commission made their recommendations, 
much has changed, but a few things have remained the same. 
The T&E community must never forget the principal reason 
for testing is to learn and to gain knowledge and 
information about the system undergoing design and 
development. No matter how "open" the system becomes, this 
need to learn remains, and testing at the lowest level, to 
the highest (operational) level is key. But testing must 
be done by experienced professionals using proven methods 
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and given adequate recourses. Is the current T&E 
infrastructure ready to handle the challenges that lie 
ahead? 

The current Director of Operational Test and 
Evaluation recently concluded his remarks on T&E Role in 
Experimentation with the following: 

We, the T&E community - in both industry and 
government, both technical and operational 
testers - have served the Department very well 
over the years. 

There is a new world dawning that calls for 
new and innovative strategies and capabilities 
for T&E. I am confident that, together, we will 
rise to the challenge as we have in the past and 
ensure that our soldiers, sailors, and airmen are 
equipped with the best eguipment our nation can 
provide (Christie, "Test & Evaluation's Role in 
Experimentation," 2002.) 
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